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that were included in the questions. Most of the words in the questions were 
correctly defined by at least 40 percent of the sample. Children who could 
define the word were also more successful in answering the corresponding 
comprehension questions. A second study determined how accurate students were 
in predicting how well they knew the words in comprehension questions. 

Twenty- two fourth graders were asked to read a story, answer comprehension 
questions, rate how well they knew specific words in the narrative, and then 
give meanings for the words. Most students, generally 90 percent of the 
sample, were accurate in their estimation of v;ord knowledge. However, the 
percentage of students who over-estimated their word knowledge increased with 
words that had a lower percentage correct on the definitions. As in the first 
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questions correctly if they knew how to define key words related to the 
question. Educational implications for actively teaching vocabulary in the 
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^ ABSTRACT 

There are few studies on the word knowledge of elementary students and none that we have 
found addressing the relationship of their knowledge of specific words to comprehending passages 
involving those words. Two studies address these issues. In the first study, a sample of Grade 5 
and 6 students (N=39) read four passages, answered comprehension questions and defined words 
which were included within these comprehension questions. Most of the words within these 
questions were correctly defined by at least 40% of the sample. As well, if children could define 
the word, they were more successful at answering the corresponding comprehension question as 
opposed to if they did not know the correct definition. A second study determined how accurate 
students are in predicting how well they knew words in comprehension questions. The sample 
consisted of 22 Grade 4 students who were asked to read a story, answer comprehension 
questions, rate how well they knew specific words from the narrative, and then give meanings for 
the words. Most students were accurate in their estimation of word knowledge, generally 90% of 
the sample. However, the percentage of students who over-estimated their word knowledge 
increased with words which had a lower percentage correct on the definitions. As with Study 1, 
students had a greater probability of correctly answering comprehension questions if they knew 
how to define key words relating to the question. Educational implications for actively teaching 
vocabulary at lower grades are discussed. 
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Metacognition of Vocabulary Knowledge: A Preliminary Study 
Hadley Koltun, M.A., and Andrew Biemiller, Ph.D. 



Department of Human Development and Applied Psychology, 

Ontario Institute for Studies in Education of the University of Toronto, 

Toronto, Canada 

Vocabulary has long been recognized as a correlate of intelligence as assessed by IQ 
(Kaufman, 1994; Sternberg, 1987; Weschler, 1991) and academic success in several areas, 
especially in the area of reading and language comprehension (Bloom, 1976). There has been an 
increase in interest in vocabulary knowledge and how it relates to text (Graves, Juel & Graves, 
1998), reviews of methods (Stahl, 1998) and research (Biemiller, 1998). 

The development of vocabulary in the primary years varies markedly and is dependent on 
several factors. One such factor is whether the same root word is used for different meanings (e.g. 
whether book, booked and rebooked are considered to be one, two, three or more separate 
vocabulary items). Using a conservative criteria, (e.g. treating book as two words in referring to 
books we read and for booking (scheduling a trip), it appears that the average child leams from 
1200 words a year or about 3 to 4 root words per day (Anglin, 1993; Biemiller, 1999). Because 
of the large number of words that must be learned, it is clearly important that children build 
vocabulary at a fairly rapid rate throughout elementary school. However, little effort is made in the 
primary grades to do so (Becker, 1977; Cantalini, 1987; Morrison et al, 1998). More emphasis on 
vocabulary is given in the upper elementary years, but this appears to have a limited effect for those 
who enter these grades with restricted vocabulary (Chall, Jacobs & Baldwin, 1990). 

There were two specific aims within this study. A first aim was to determine the agreement 
between the students’ self-report and the open ended definitions of the same word.The second aim 
was the investigation of the relationship between self-reports of vocabulary knowledge within a 
passage and the successful answering of comprehension questions (with the same word in the 
question). 

In this paper, one study is reported of students’ abilities to tell what words they know. Self 
knowledge of words is critical if students are to either actively infer meanings of unfamiliar or little 
known words on their own or to seek assistance from others regarding word meanings. Both 
Beck & McKeown ( 1991) and Graves, Juel and Graves ( 1998) emphasize the importance of word 
consciousness or word awareness in building vocabulary. The word conscious student is one who 
is interested in words, recognizes the correct usage of words and is vigilant for new meanings of 
words (Graves, Juel & Garves, 1998). Word consciousness is viewed as an important goal of 
vocabulary instruction and it is understood to ben an outcome of successful vocabulary instruction 
(Beck & McKeown, 1991; Stahl & Fairbanks, 1986; Graves, Juel & Graves, 1998). Indirect 
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evidence for this hypothesis is provided by the finding that one outcome of vocabulary instruction 
is measurable vocabulary gains, as compared to non-instructed controls, on words not included in 
the instruction (Stahl & Fairbanks, 1986). 

It is understood that the word conscious student would have an accurate estimate of his or her 
own word knowledge. Curtis (1987) indicates that self-reports of specific vocabulary tends to 
overestimate word knowledge, but bear some relationship to vocabulary as assessed in other ways, 
Children at the grade 5 level showed some knowledge of how well they knew words, but their 
reports were not very accurate. We have found no studies have looked at the relationship between 
the word knowledge of elementary school students and their reading comprehension using those 
words. 



Study 1; 

Subjects: 

Thirty-nine Grade 5 and 6 children in a laboratory school at the University of Toronto 
participated in this study (18 Grade 5 and 21 Grade 6 students). 



Materials: 

Canadian Test of Basic Skills (CTBS)-Comprehension Section (King, 1982). Four stories 
(Circus Salad Story, Rusty Story , Lightning Bug Story, and Chinese Kite Story) were used to 
assess comprehension in the students from the laboratory school. The readability of each of these 
stories (new Dale-Chall readability coefficient) were calculated using Readability Master 2000 
software. The respective Dale-Chall readability grade levels were 3 for Circus Salad and Rusty, 
and 4 for Lightning Bug and Chinese Kite. 

Each of the four stories were copied into the survey used in this study. Students were required 
to read the story and answer the comprehension questions. On the following pages, the text of the 
narrative was repeated along with the vocabulary which was assessed as being potentially more 
difficult for these students. Most of the words were identified by Readability Master as being 
potentially difficult for these students (i.e. those words not included on the list of 3000 words). 
Children were required to define each of these words in writing. The story was reproduced below 
these words with the vocabulary words in bolded lettering for easy identification. 



Procedure: 



The thirty-nine students were presented with the four stories in the form of an inventory called 
Understanding Language and Vocabulary. They were asked to read the stories and complete the 
comprehension questions before turning to the next page, which contained the open-ended 
vocabulary words which the students were required to define. 

Results and Discussion: 

Two judges rated whether the open-ended vocabulary words were correctly defined relative to 
the context by using the following criteria: one point was given if the judge believed the word was 
correctly defined, half-a-point if it were believed that the student may know what the word meant 
and zero points if the judge believed that the child did not know what the word meant. The 
ratings of the open-ended definitions were highly reliable (r=.98) 

As it has been pointed out, rating words as either known or not known is problematic. As 
children and adults get older, understanding of concepts become more sophisticated. The general 
principle used in this study was whether the child’s definition in our best estimate would allow 
them to understand a narrative which incorporated the test sentence. For example, if a child wrote 
that divided meant "to cut into sections” that definition would be given a full point since she 
communicated that she understands the word. If she had written “tocut”, she may know the 
meaning of the word but she has not demonstrated it fully, so she would be awarded half-a-point. 
If she would have written that divided meant “to share”, then she would have been awarded zero 
points. The authors of this study preferred a more liberal approach to whether a word was known 
or not known, as opposed to the more exacting principles set out by Weschler (1991) for the 
Vocabulary subtest of the WISC-III. 

The percentage correct (as a percentage of the total sample, N=39) for each of the words used 
in the survey are shown in Table 1. The distribution of the percentage correct in the each of the 
grades is shown in parentheses. In the Lightning Bugs Story, the percentage correct is above 90% 
with the exception of the word batting (82%). Words for the remaining three stories tended to be 
more difficult for both grades. However, the majority of words are above the 40% of the entire 
sample, with the exception of the word holed (36%), which is an idiom. 

The comprehension questions used in the survey Understanding Ixmguage and Vocabulary 
were drawn directly from the CTBS and the correct scoring was determined by the answer key of 
the testing manual. The percentage correct of these questions are shown in the descending order in 
Table 2. The easier questions, yielding higher percentages correct, were generally factual 
comprehension questions and could be answered by referring back to the text (e.g.C9 Why are 
children batting at the air?). Questions which yielded the lowest percentages correct were 
inferential questions which require the student to draw logical conclusions based on facts or to 
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demonstrate knowledge of the gist of the narrative (e.g. F25 What is the author trying to 
show in the second paragraph?). 

Table 2 also shows the words which students were asked to define and the how successful 
they could be if they either knew how to define the word or if they did not know how to define the 
word. For the most part, if they could define the word, then the children tended to answer the 
corresponding comprehension question correctly. As above, the higher rates corresponded with 
factual questions and the lower rates with inferential comprehension questions. Successful 
answering of comprehension questions involving inferences may be dependent upon factors above 
and beyond vocabulary knowledge. However, if vocabulary knowledge is weak, this will lead to 
a lower probability of correctly answering an inferential comprehension question. 

The first study only addresses word knowledge in the stories presented and does not address 
how accurate these students are in predicting how well they know these words. A necessary 
component of word consciousness is how successful students are in predicting if they could define 
a word. Thus, the second study was designed to address the issue of how accurately children can 
estimate their own vocabulary knowledge and how their estimates are related to their reading 
comprehension. 



Study 2; 

Subjects: 

Twenty-two Grade 4 students at the same laboratory school as Study 1 participated in this 
study. 

Materials and Procedure: 

Canadian Test of Basic Skills (CTBS)-Yellowstone Story: Comprehension Section (King, 

1982 ). The Yellowstone Story from the CTBS has been assessed at a reading grade level of 9-10 
(new Dale-Chall readability coefficient). This story was chosen since the vocabulary demand was 
several grade levels above the children’s grade 4 level. The advanced vocabulary increased the 
chances that students would not know certain words presented to them in the narrative. As in 
Study 1, students were first required to read the story and answer a series of comprehension 
questions. On the next page, students were asked to indicate how well they knew words. 

Students were instructed to indicate how well they knew the words which were bolded in the 
narrative. Students were instructed to rate the word in the following categories: They had never 
seen the word before (do not know the word); have seen the word but did not know what the word 
meant; know what the word means but do not use the word regularly; or know what the word 
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means and use it on a regular basis. To make sure that the students were not falling into a response 
set, several pseudo-words (e.g. laift ) were implanted and bolded in the narrative. Students were 
asked to rate these words as indicated above. On the final two pages, the narrative was reproduced 
for the third time and students were asked to define the bolded words within the narrative as they 
had done in Study 1. 



Results and Discussion: 

Table 3 shows the percentage correct of the defined words by the ratings of the corresponding 
words (10 words). The overall percentage correct of the definitions indicates a range of difficulty 
of words with petriDed, accumulate and geologist being the most difficult words to define. 
However, the ratings of word knowledge as assessed by this method indicates that students at least 
had some knowledge of these words. Finally, students were accurate in identifying that they did 
not know what the pseudo-words meant, indicating that they were not simply claiming to know all 
the words. 

In order to understand the accuracy of these students’ resjjonses, it was necessary to look at the 
agreement between the word ratings and the corresponding open-ended definitions of the same 
word. This effort resulted in the development of the following matrix as shown in Table 4. This 
matrix outlines how accurately children estimated their own ability to define words: 
Under-estimators were those students who rated their knowledge of a specific word lower than 
their ability to define the word; Accurate Estimators were those students who matched their rating 
of their knowledge of a specific word with their ability to define that word; and Over-Estimators 
were those students who rated their knowledge of a word above that of their ability to define the 
word. These classifications are shown in Table 4. 

Table 5 addresses each word in the Y ellow stone stoiy which was both rated and defined. The 
number and the percentage of under-estimators remained the same across the words as was the 
case for the accurate estimators (with the exception of the words, petriDed, accumulate and 
geologist, where both the number and percentage decreased far below 90%). However, for 
those same words, the number and percentages of over-estimators increased. This finding may 
suggest that these students may over-estimate their ability to define words when they are more 
difficult. 

The final question to answer is what is the relationship between accuracy of word knowledge 
and the ability to comprehend the narrative. Table 6 shows the difficulty level of each 
comprehension question along with the key words in each question which were on the survey. Of 
greater interest is the percentage correct if the students knew how to define the key word and the 
percentage correct if they did not know how to define the key word. With some exceptions. 
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knowing the meaning of a word corresponded with a greater probability of answering the 
comprehension question correctly. As with Study 1, the largest differences between those who 
could successfully answer the comprehension questions were for inferential comprehension 
questions. Of interest were those questions and key words in which the predicted relationship was 
in the wrong direction. For example, the word volcano in question A3, a question which asked 
for the definition of fossil. The word volcanic was not a correct option so it was not important 
for a child to know the word volcano in this context. 



General Discussion: 

In both studies, elementary grade children were generally more successful in answering 
comprehension questions if they knew the meaning of key words in the question. However, even 
if they knew the correct meaning of the word, it would not guarantee that the students could 
answer the questions correctly. This observation became more salient when factual and inferential 
comprehension questions were compared; there are factors other than vocabulary knowledge which 
account for the correct answering of inferential comprehension questions, such as the ability to 
keep disparate facts in short-term memory and the ability to reason. 

The second study added information on the metacognitive ability of students to assess their 
own knowledge. It appears that students can accurately assess word knowledge when words are 
of easy to moderate levels of difficulty. In those instances where words are generally difficult for 
the population studied, students tend to over-estimate their own ability to define words. Since the 
sample was small enough, we were able to track some of the students who overestimated their 
word knowledge and found they consistently gave an incorrect response to the specific 
comprehension question. This clinical analysis emphasizes the need to for teachers to address and 
teach the skill of accurately identifying word knowledge. The second study emphasizes the need to 
help students assess their own word knowledge when reading material is more difficult than their 
frustration level of reading. 

The second study indicates that self-reports of word knowledge by themselves can be 
unreliable. It is important to use other methods to assess vocabulary knowledge, such as 
definitions of key words and the teaching children how to figure out word meanings. Another 
factor which may improve upon the accuracy of word knowledge is to remove the focus of the 
child to the word by asking them if the word is one which other students may find difficult. This 
survey asked each child directly which may be difficult for at least two reasons; it asks a child to 
admit to his own short-comings in word knowledge and it may be more difficult for a student in 
elementary school to observe his own process in word knowledge than to observe others, which 
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he has probably been doing for some time (Gordon Wells, personal communication). 

The use of self-report methods of vocabulary knowledge may become useful once students’ 
word knowledge and their metacognition about word knowledge increases. Teachers and 
educational researchers alike should be aware of the developmental process of word consciousness 
in children. Future studies should look at this process across various ages and grades with a 
variety of graded reading material. Methods of assessment of word consciousness should not be 
restricted to self-report but should also include open-ended definitions and more standardized 
approaches to identifying various types of comprehension questions (i.e. factual versus inferential 
comprehension questions). Teachers can use this corpus of research to become aware of the need 
to actively assess and teach vocabulary in the lower grades. Teaching vocabulary is not just for the 
sake of improving expressive vocabulary but also serves to improve listening and reading 
comprehension, the ultimate goals of literacy. 



TABLE 1; Study 1: Percentage Correct for Word Definitions for Total Sample 



Word 


Percentage Correct 
(Grade 5, Grade 6) 






Batting 


82 (89, 76) 


Lightning Bug 


90 (78, 100) 


Wingless 


95(100, 90) 


Member 


90 (88, 90) 


Similar 


95 (89, 100) 


Larger 


97 (94, 100) 


Dim 


90(83,95) 


Flickering 


90 (83, 95) 


Shelled 


46 (56, 38) 


Nut Grinder 


95 (94, 95) 


Prepare 


56 (50, 62) 


Describe 


90 (94, 86) 


Peel 


92 (89, 95) 


Divided 


82 (78, 86) 


Rodeo 


54 (44, 62) 


Holed 


36 (39, 33) 


Supplies 


87(83, 90) 


Catalogue 


64(56,71) 


Dragon 


85(83, 86) 


Scales 


56 (44, 67) 


Moveable 


87 (78, 95) 


Recreation 


44(33,52) 


Unusual 


97 (94, 100) 


Armed Forces 


90 (89, 90) 


Afloat 


90 (89, 90) 


Signals 


90 (72, 67) 


Rescue 


95 (94, 95) 


Entertainment 


46 (39, 52) 
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Word 


Percentage Correct 
(Grade 5, Grade 6) 






Batting 


82 (89, 76) 


Shipwrecked 


77(61,90) 


Carriers 


41 (39, 43) 


Rugged 


56 (50, 62) 


Attention 


59 (39, 76) 



TABLE 2: Overall Percentage Correct for Comprehension Questions (study 1) and 
Percentage Correct Dependent upon Word Knowledge (N=39) (number of subjects 
in parentheses). 



Question 


Percentage 

Correct 


Word 




Percentage 
Correct for 
Comp. 
Questions 


Prediction 








If children 
knew Word 


If children 
did not 
know word 
















D13 


100 


Peeled 


92(36) 


100(3) 


Little diff. 






Divided 


82(32) 


100(7) 


WW 


D12 


97 


Describe 


97(32) 


100(7) 


Little diff 


C9 


95 


Batting 


94(30) 


100(7) 


Little diff 


CIO 


95 


Lightning Bugs 


97(34) 


100(7) 


Little diff 


Cll 


95 


Lighting Bugs 


100(35) 


50(2) 


RW 






Flickering 


97(34) 


75(3) 


RW 


D14 


95 


- 








E19 


90 


Supplies 


91(31) 


80(4) 


RW 






Catalogue 


96(24) 


79(11) 


RW 


F21 


90 


Dragon 


91(30) 


83(5) 


RW 






Scales 


95(21) 


82(14) 


RW 


















Moveable 


91(31) 


80(4) 


RW 


E17 


85 


- 








E16 


79 


- 








F22 


79 


Unusual 


82(31) 


0 


RW 


E18 


77 


Supplies 


77(26) 


80(4) 


Little diff 


F20 


69 


Recreation 


94(16) 


50(11) 


RW 


F23 


67 


Afloat 


66(23) 


75(3) 


WW 






Signals 


67(18) 


67(8) 


Little diff 


















Rescue 


68(25) 


50(1) 


RW 






Entertainment 


78(14) 


57(12) 


RW 
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Question 


Percentage 

Correct 


Word 




Percentage 
Correct for 
Comp. 
Questions 


Prediction 








If children 
knew Word 


If children 
did not 
know word 








Shipwrecked 


67(20) 


67(6) 


Little diff 


F25 


67 


- 








F24 


51 


Carriers 


69(11) 


39(9) 


RW 






Rugged 


64(14) 


35(6) 


RW 






Attention 


57(13) 


44(7) 


RW 
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Table 3: Study 2: Percent Correct Definitions by Ratings for 10 Words (number 
of children in parentheses) N=22 



%(n) 


Not Known 


Seen But 
Did Not 
Know 


Know But 
Not Used 


Know and 
Used 


Overall Correct 
Definitions 






































Strange 






100(5) 


82(17) 


86(19) 


Volcano 


100(1) 




77(13) 


88(8) 


82(18) 


Remarkable 


100(1) 


0(1) 


93 (15) 


80(5) 


86(19) 


Fossil 




0(1) 


79(19) 


50(2) 


73(16) 


Accompany 


0(1) 


20(1) 


83 (12) 


75 (4) 


64(14) 


Petrified 


0(1) 


10(11) 


44(9) 


0(1) 


23(5) 


Accumulate 


0(8) 


0(9) 


50(4) 


0(1) 


9(2) 


Geologist 


0(2) 


0(5) 


29(14) 


0(1) 


18(4) 


























Pseudo-words 
























Hets 


0(19) 


0(2) 






0 


Shif 


0(17) 


0(1) 


0(4) 




0 



13 




14 



Table 4: Agreement Between Word Ratings and Open-Ended Deflnitions Matrix. 





Open Ended Definitions 


Self-Report Ratings 


Doesn't Know 
Word 


May Know 
Word 


Knows Word 






















Not Seen/Not Known 


Accurate 


Under-Estimate 


Under-Estimate 


Seen Word/Don't Know Word 


Accurate 


Accurate 


Under-Estimate 


Know Word/ Don't Use Word 


Over-Estimate 


Accurate 


Accurate 


Know and Use Word 


Over-Estimate 


Over-Estimate 


Accurate 



* Table 5: Percentage of Word DeHnitions Correct as a Function of the Accuracy of 
the Students' Ratings of Word Knowledge (percentage of cases in parentheses) (N=22) 







Classification 




Real Words 


Under Estimators 


Accurate 


Over Estimators 










Strange 


0 


21 (95) 


1(5) 


Volcano 


1(5) 


20(91) 


1(5) 


Remarkable 


1(5) 


20(91) 


1(5) 


Fossil 


0 


21 (95) 


1(5) 


Accompany 


1(5) 


20(91) 


1(5) 


Petrified 


1(5) 


15(68) 


6(27) 


Accumulate 


0 


20(91) 


2(9) 


Geologist 


1(5) 


14(64) 


6(27) 
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Table 6: Percentages of Students Who Could Successfully Answer the 
Comprehension Question as a Function of Whether They Knew the Word or Not 

(N=22) 



Question 


Percentage 

Correct 


Word 


Percentage Correct on 
Comprehension Question if 










Children 
Knew Word 


Children did 
not know 

word 


Expected 

Difference 


A2 


45 


Volcano 


44 


50 


WW 


A3 


45 


Fossil 


60 


12 


RW 






Volcano 


39 


75 


WW 


A4 


36 


Fossil 


50 


0 


RW 






Petrified 


60 


29 


RW 


A5 


45 


Volcano 


56 


0 


RW 






Petrified 


60 


41 


RW 


A6 


18 


Fossil 


19 


17 


Little diff 






Petrified 


20 


18 


Little diff 


A7 


18 


Fossil 


25 


0 


RW 






Petrified 


40 


12 


RW 
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